Papers by Yash Kumar Lal
CaT-Bench: Benchmarking Language Model Understanding of Causal and Temporal Dependencies in Plans (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on reasoning in plans focus on classical problems, simulated environments, or restricted language such as PDDL, but real-world plans cannot be tested to test for correctness and reliability. |
| Approach: | They propose a benchmark question that tests whether a step must necessarily occur before or after another in cooking recipe plans. |
| Outcome: | The proposed question-driven evaluation shows that SOTA LLMs are underwhelming and biased towards predicting dependence more often, but the best F1 result is 0.73. |
Using Commonsense Knowledge to Answer Why-Questions (2022.emnlp-main)
Copied to clipboard
Yash Kumar Lal, Niket Tandon, Tanvi Aggarwal, Horace Liu, Nathanael Chambers, Raymond Mooney, Niranjan Balasubramanian
| Challenge: | Existing approaches to integrating commonsense knowledge into large language models are implicit and explicit. |
| Approach: | They analyze the effects of model size and methods of injecting knowledge into TellMeWhy datasets to determine what aspects of commonsense knowledge are available in large language models. |
| Outcome: | The largest models yield substantial improvements over base models, but the amount of improvement decreases with larger model size. |
IrEne-viz: Visualizing Energy Consumption of Transformer Models (2021.emnlp-demo)
Copied to clipboard
Yash Kumar Lal, Reetu Singh, Harsh Trivedi, Qingqing Cao, Aruna Balasubramanian, Niranjan Balasubramanian
| Challenge: | IrEne is an energy prediction system that accurately predicts inference energy consumption of transformer-based NLP models. |
| Approach: | They present an online platform for visualizing and exploring energy consumption of transformer-based NLP models. |
| Outcome: | The proposed system predicts energy consumption of transformer-based models and their components. |
De-Mixing Sentiment from Code-Mixed Text (P19-2)
Copied to clipboard
| Challenge: | Code-mixing is the phenomenon of mixing the vocabulary and syntax of multiple languages in the same sentence. |
| Approach: | They propose a hybrid architecture for the task of Sentiment Analysis of English-Hindi code-mixed data using CNNs to generate subword representations for the sentences. |
| Outcome: | The proposed architecture achieves 83.54% accuracy and 0.827 F1 score on a benchmark dataset. |
SOCIALITE-LLAMA: An Instruction-Tuned Model for Social Scientific Tasks (2024.eacl-short)
Copied to clipboard
Gourab Dey, Adithya V Ganesan, Yash Kumar Lal, Manal Shah, Shreyashee Sinha, Matthew Matero, Salvatore Giorgi, Vivek Kulkarni, H. Schwartz
| Challenge: | Social science NLP tasks require large data to capture semantics and implicit pragmatics. |
| Approach: | They propose an open-source instruction tuning tool for social science NLP tasks that captures implicit pragmatic cues from text. |
| Outcome: | The proposed model matches or improves on a state-of-the-art, multi-task finetuned model on 80% of social tasks. |
Tailoring with Targeted Precision: Edit-Based Agents for Open-Domain Procedure Customization (2024.findings-acl)
Copied to clipboard
| Challenge: | Using a set of over 200 WikiHow procedures, we test several simple multi-LLM-agent architectures for customization. |
| Approach: | They propose to use a set of WikiHow procedures to test how-to procedures can be customized by multiple LLMs. |
| Outcome: | The proposed architecture outperforms an end-to-end LLM in the evaluation set of over 200 WikiHow procedures. |
TellMeWhy: A Dataset for Answering Why-Questions in Narratives (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing models do not have the ability to answer "why" questions that require commonsense knowledge external to the narrative. |
| Approach: | They propose a crowd-sourced dataset that asks why characters perform actions . they show that state-of-the-art models are far below human performance on answering such questions . |
| Outcome: | The proposed dataset shows that state-of-the-art models are far below human performance on answering such questions. |
Evaluating Paraphrastic Robustness in Textual Entailment Models (2023.acl-short)
Copied to clipboard
| Challenge: | Recognizing Textual Entailment models understand language and should be robust to paraphrases. |
| Approach: | They propose to evaluate whether RTE models are robust to paraphrase . they use 1,126 pairs of Recognizing Textual Entailment (RTE) examples to evaluate their models . |
| Outcome: | The evaluation set shows that models change predictions on 8-16% of paraphrased examples, suggesting that there is room for improvement. |
IrEne: Interpretable Energy Prediction for Transformers (2021.acl-long)
Copied to clipboard
| Challenge: | Existing software-based energy measurements of NLP models are not accurate because they do not consider the complex interactions between energy consumption and model execution. |
| Approach: | They propose an interpretable and extensible energy prediction system that predicts inference energy consumption of Transformer-based NLP models. |
| Outcome: | The proposed system predicts inference energy consumption of transformer models with an error of under 7% compared to the ground truth. |
SAGEViz: SchemA GEneration and Visualization (2023.emnlp-demo)
Copied to clipboard
Sugam Devare, Mahnaz Koupaee, Gautham Gunapati, Sayontan Ghosh, Sai Vallurupalli, Yash Kumar Lal, Francis Ferraro, Nathanael Chambers, Greg Durrett, Raymond Mooney, Katrin Erk, Niranjan Balasubramanian
| Challenge: | Schema induction involves creating a graph representation depicting how events unfold . supervised and few-shot approaches are not scalable and time-consuming . |
| Approach: | They propose a tool that utilizes human-AI collaboration to create and update complex schema graphs efficiently. |
| Outcome: | The proposed tool can generate schemas of better quality and be used by users in a variety of domains. |
Temporal Reasoning in Natural Language Inference (2020.findings-emnlp)
Copied to clipboard
| Challenge: | We use five new natural language inference (NLI) datasets focused on temporal reasoning. |
| Approach: | They introduce five new natural language inference datasets focused on temporal reasoning. |
| Outcome: | The proposed models capture the temporal reasoning of four existing datasets. |